1. Field of the Invention
This invention relates to optical metrology, and particularly to the problem of making accurate non-contact dimensional measurements of objects that are viewed through an endoscope.
2. Description of Related Art
2a. Perspective Dimensional Measurements with Endoscopes
In the past several decades, the use of optical endoscopes has become common for the visual inspection of inaccessible objects, such as the internal organs of the human body or the internal parts of machinery. These visual inspections are performed in order to assess the need for surgery or equipment tear down and repair; thus the results of the inspections are accorded a great deal of importance. Accordingly, there has been much effort to improve the art in the field of endoscopes.
Endoscopes are long and narrow optical systems, typically circular in cross-section, which can be inserted through a small opening in an enclosure to give a view of the interior. They almost always include a source of illumination that is conducted along the interior of the scope from the outside (proximal) end to the inside (distal) end, so that the interior of a chamber can be viewed even if it contains no illumination. Endoscopes come in two basic types; these are the flexible endoscopes (fiberscopes and videoscopes) and the rigid borescopes. Flexible scopes are more versatile, but borescopes can provide higher image quality, are less expensive, are easier to manipulate, and are thus generally preferred in those applications for which they are suited.
While endoscopes (both flexible and rigid) can give the user a relatively clear view of an inaccessible region, there is no inherent ability for the user to make a quantitative measurement of the size of the objects he or she is viewing. There are many applications for which the size of an object, such as a tumor in a human body, or a crack in a machine part, is a critically important piece of information. Making a truly accurate measurement under these circumstances is a long-standing problem that has not been adequately solved until recently.
In a first co-pending application, now U.S. Pat. No. 6,009,189, entitled “Apparatus And Method For Making Accurate Three-Dimensional Size Measurements Of Inaccessible Objects”, filed Aug. 16, 1996, and which is incorporated herein by reference, I taught a new and complete system for making measurements of objects with an imaging optical system, with particular emphasis on endoscopic applications. In a second co-pending application, now U.S. Pat. No. 6,121,999, entitled “Eliminating Routine Alignment Calibrations in Perspective Dimensional Measurements”, filed Jun. 9, 1997, I taught certain improvements to the measurement system as applied to the endoscopic application. I will hereinafter refer to the first co-pending application as “Application 1” and the second as “Application 2”.
My previous invention makes possible a new class of endoscopic measurement instruments of unprecedented measurement accuracy. This measurement system is a version of a technique I call “perspective dimensional measurement”. By “perspective” I am referring to the use of two or more views of an object, obtained from different viewing positions, for dimensional measurement of the object. By “dimensional measurement”, I mean the determination of the true three-dimensional (height, width, and depth) distance(s) between two or more selected points on the object.
As a necessary and integral part of my complete measurement system, I taught how to calibrate it in the referenced applications. I taught the use of a complete set of robust calibration procedures, which removes the need for the measurement system to be built accurately to a specific geometry, and also removes any need for the imaging optical system(s) to be built accurately to specific optical characteristics. Instead, I taught how to calibrate the geometry and characteristics of the opto-mechanical hardware, and how to take that actual geometry into account in the measurement process. The complete set of calibration procedures I taught includes three different types of calibration. In optical calibration, the detailed characteristics of each imaging optical system (i.e., camera), when used as a precision image forming device, are determined. In alignment calibration, the orientations of each camera's measurement coordinate axes with respect to the motion of the camera are determined. Finally, in motion calibration, any errors in the actual motion of the camera(s), as compared to the ideal motion, are determined.
In Application 2, improvements to the system were made that eliminated the necessity of repeating the alignment calibration in certain important circumstances.
In some embodiments, my previous invention enables one to make accurate measurements using a standard, substantially side-looking, rigid borescope. Since the person who needs the measurement will often already own such a borescope, the new method offers a significant cost advantage over earlier measurement techniques.
In other embodiments, my previous invention provides for new types of self-contained endoscopic measurement instruments, both rigid and flexible, which offer significantly improved measurement accuracy as compared with those previously available. I call these new instruments the electronic measurement borescope and the electronic measurement endoscope.
While my system, as previously disclosed, does produce accurate dimensional measurements, there is room for improvement. The problem is that a new optical calibration may have to be performed each time the focus of the instrument is adjusted. One of the parameters determined during optical calibration is proportional to the magnification of the image. Without making special provisions for it, the magnification will most likely not be constant with focus, and thus every time the instrument is refocused, there is the logical requirement for a new optical calibration. Additional parameters that are determined during optical calibration are the location of the optical axis on the image sensor and the distortion of the image. These parameters may also vary as the focal state (that is, the object plane that is in focus) of the instrument is changed, which would be additional reasons to require a new calibration. Of course, whether a new calibration would actually be required in any specific instance depends on the accuracy required of the dimensional measurement, and on the characteristics of the camera being used in that instance.
When a standard borescope is used with my previous invention, there is the further difficulty that when an image sensor is mounted to the borescope to perform perspective measurements and the assembly is then calibrated, this calibration is lost if the image sensor is subsequently removed from the borescope. One may wish to remove the image sensor temporarily either to use the borescope for visual inspection, or to use the image sensor with another borescope that has different characteristics. What is needed here is a way to allow the removal and replacement of the measurement image sensor while maintaining calibration of the measurement system.
2b. Magnification and Focus in Optical Metrology and Machine Vision
It is known that the magnification of an image formed by an optical system depends on the range of the object; that is, the magnification depends on the distance between the object and the optical system. It is also known that there is a well defined relationship between the position of a focusing component in an optical system and the range of an object that is in focus. These known relationships have been used in a class of endoscopic measurement instruments that implement a technique that I call measurement by focus. U.S. Pat. No. 4,078,864 to Howell (1978) and U.S. Pat. No. 5,573,492 to Dianna and Costello (1996) are examples of this approach. In these instrunents, a focusing component is instrumented to produce a datum that is a function of the range to the object plane that is in focus. (By “instrumented”, I simply mean that the position of the focusing component along the system's optical axis is measured with respect to some fixed reference within the system.) This datum is then used together with a calibrated relationship between the range and the magnification to produce a relationship between dimensions as measured on the image and the corresponding dimensions on the object.
The fundamental problem with this approach to making accurate dimensional measurements is that the position at which the image is in focus is difficult to determine, thus the range measurement, and hence the magnification, is subject to relatively large random errors (that is, a lack of repeatability). As a result, these instruments, as disclosed and in practice, are restricted to making two dimensional measurements, i.e., measurements of distances that are oriented perpendicular to the optical axis.
The key assumption of the measurement by focus technique is that there is a fixed relationship between the range of an object and its magnification, because it is assumed that an object will always be viewed in focus. This assumption cannot apply to any three-dimensional measurement technique because the measurement determines the depth of an object, while only a single plane of the object can truly be in focus at a time. In general the magnification of an image depends not only on the range of the object, but also on the range at which the optical system is focused, that is, on the focal state of the optical system.
The telecentric principle has often been applied to standard two dimensional optical measurements, such as those made by optical comparators. Telecentricity refers to the situation where the cone of light forming each point in an image has a central axis which is parallel to the optical axis of the system. It is only recently that telecentricity has been applied to three-dimensional machine vision applications, in a paper by M. Watanabe and S. K. Nayar: “Telecentric Optics for Computational Vision”, Lecture Notes in Computer Science, 1065, 1996. The applications that these authors consider are called “depth from focus” and “depth from defocus” and are closely related to the measurement by focus technique.
In depth from focus, the focal state of the imaging camera is varied in small steps throughout a range of focal states. An image of the scene of interest is acquired at each of these focal states. The images are then analyzed to determine in which image the individual elements of the scene are in best focus, thus obtaining a relatively crude estimate of the range of each element of the image. In depth from defocus, only two images are acquired at two focal states, and a different image processing scheme is used in an attempt to obtain the same information. As stated, these are machine vision applications, where the goal is a relatively coarse determination of the three-dimensional layout of the objects in a scene, rather than metrology, where the goal is a precise measurement of individual dimensions on an object. These techniques have the same problem as does the measurement by focus technique when it comes to accurate metrology.
Watanabe and Nayar point out that when a telecentric optical system is used and the image viewing plane is moved to focus the system, the magnification of the image does not vary with the focal state of the system. While they refer to this situation as “constant magnification”, I prefer to call it “constant relative magnification”. The first reason for defining a new term is that this is not the only type of constant magnification; there are other types of constant magnification which are important. Significant additional reasons for the use of my terminology will become apparent in the ensuing discussion.
While the system taught by Watanabe and Nayar meets the goals they set for it, their teachings are far from complete. As I will show, there are significant benefits to be gained from going beyond the use of a telecentric optical system for optical metrology. In addition, their teachings are in error in an important point, in that they state that their system will also work if the lens is moved with respect to the image viewing plane. I will show that this is incorrect in practice, and that the difference is important to accurate metrology.
In U.S. Pat. No. 4,083,057 (1978), Quinn disclosed the addition of a magnification corrector to an auto-focus lens system to correct for the change in magnification with focal state in video systems. The problem addressed by Quinn was that objects near the edge of the field of view of a video or movie camera would be seen to move into or out of the field of view as the focus of the camera was changed. At first glance, this problem is not related to machine vision or metrology, but Quinn's system is the earliest example known to me of a non-telecentric system that may be able to achieve imaging at constant relative magnification. I say “may be able to” because Quinn does not teach everything that is required to generate a correction which will work for objects at all ranges. I am not aware of any later work which remedies this deficiency.
A system that enables one to make photographs at constant magnification was disclosed by Yasukuni, et. al., U.S. Pat. No. 4,193,667 (1980), and has been followed by many other patents directed to the same end. In these systems, the goal is to image an object at a constant image size as that object is moved to various ranges, while also keeping the image in focus as it moves. I refer to this goal as imaging at constant absolute magnification. To perform this function, these systems use a variable focal length optical system combined with a focus adjusting component. Thus, these systems are adaptations of zoom or varifocal lenses.
A system for photocopiers, microform readers, and the like, which allows one to accurately set a variable magnification of the image of a fixed object, while simultaneously keeping the image in focus, was disclosed by Sugiura, et. al. in U.S. Pat. No. 4,751,376 (1988). This also has been followed by many other patents directed to the same end. These systems are also directed toward achieving a constant, and well-determined, absolute magnification. In this case, the focal length of the optical system is fixed, and the optical system is moved with respect to the object in order to change the magnification.
In all of the known systems providing constant absolute magnification, and in the known non-telecentric systems which may provide constant relative magnification, there are used two independently moving components in the optical system. These moving components are a magnification adjuster and a focus adjuster. The magnification adjuster is typically a lens group, while the focus adjuster is either a second lens group or an optical path length adjusting component. The relative motions of these components with respect to the object and with respect to each other are then controlled in a manner to produce the desired magnification and focus result. In the early systems described by Yasukuni, et. al. and by Quinn, the relative motions are controlled by mechanical cams. In later devices, motors are used to move the components, and position transducers are used to monitor the positions of the components.
For the purposes of endoscopic dimensional measurements, the fixed focal length constant absolute magnification systems are not applicable, since the distance between the object and the optical system cannot, in general, be controlled to adjust the magnification. The variable focal length systems could be applied to these measurements as well as to general optical metrology, but systems using two instrumented moving components are complex and expensive. In addition, because the disclosed systems have been designed for photography and not metrology, they do not produce the information necessary for accurate metrology.
In the general art of optical metrology, little attention has been paid to the detailed characteristics of the magnification in out of focus images, other than the use of the telecentric principle. It has not been well understood that accurate measurements can be made with out of focus images. The only related reference that I am aware of is an early letter by W. Wallin, “A Note on Apparent Magnification in Out-of-Focus Images”, Journal of the Optical Society of America, 43, 60, 1953. In this paper Wallin states that the magnification is defined only for the image plane, but then goes on to define what he calls an “apparent magnification” for out of focus images. Wallin's comment is clearly incorrect, as any real image will almost never be in perfect focus, yet such images can be used for metrology. Wallin gives expressions for the apparent magnification of an optical system in a fairly general, obscure, and unusable form. He states that these expressions are useful for tolerancing optical comparators. What this paper lacks is any information or insight into how to design or improve the design of a metrological optical system using his concept of apparent magnification.
In the general art of perspective dimensional measurements, typically the object to be inspected is brought to an inspection station that has a set of fixed viewing cameras. Thus, the range of the object is essentially fixed, and so are the focal states of the measurement cameras. If the cameras are refocused for some reason, then the system is recalibrated. Although this system is satisfactory for many purposes, I believe that if the capability to refocus the cameras on an object of interest without also requiring a recalibration were available, it would be found useful in some of these standard inspection setups.
The most important alternative to perspective dimensional measurements in three-dimensional optical metrology is known as use of “structured light”. There are, for instance, commercially available endoscopic measurement systems based on this principle. These systems have the same problem with requiring recalibration if the camera is refocused, and would also benefit if this problem were solved.
Fundamentally, what is needed for the purposes of three-dimensional optical metrology, including perspective dimensional measurements, is a way to determine the magnification that applies at each point in an image, regardless of whether that point happens to be in focus or not; and this magnification must be determined as the focal state of the optical system changes. In addition, and especially in endoscopic applications, one must also determine any deviation of the optical axis and any change in the distortion in the image as the focal state is changed. This problem has heretofore not been addressed in a comprehensive or coherent manner.
In order to meet these goals with a fixed focal length optical system, one must determine how the magnification varies with two independent variables: the range of the object point and the focal state of the optical system. With a variable focal length system, there are three independent variables: the range, the focal state, and the focal length of the system. None of the prior art systems is capable of providing the required information about how the magnification, the position of the optical axis, and the distortion depend on the independent variables, nor can they incorporate any such information into the measurement. Thus, the teachings of the prior art are not sufficient to enable one to perform accurate three-dimensional metrology while also allowing an adjustment of focus to best view an object of interest, without also requiring a recalibration when the focus is adjusted.